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FOREWORD 


Among  the  responsibilities  assigned  to  the  Office  of  the  Manager,  National 
communications  System,  is  the  management  of  the  Federal  Telecommunication 
Stanaaras  Program.  Under  this  program,  the  NCS,  with  the  assistance  of  the 
Federal  Telecommunication  Standards  Committee  identifies,  develops,  and 
coordinates  proposed  Federal  Standards  which  either  contribute  to  the  inter¬ 
operability  of  functionally  similar  Federal  telecommunication  systems  or  to  the 
achievement  of  a  compatible  and  efficient  interface  between  computer  and 
telecommunication  systems.  In  developing  and  coordinating  these  standards,  a 
considerable  amount  of  effort  is  expended  in  initiating  and  pursuing  joint 
standards  development  efforts  with  appropriate  technical  committees  of  the 
Electronic  Industries  Association,  the  American  National  Standards  Institute, 
the  International  Organization  for  Standardization,  and  the  International 
Telegraph  and  Telephone  Consultative  Committee  of  the  International 
Telecommunication  Union.  This  Technical  Information  Bulletin  presents  an 
overview  of  an  effort  which  is  contributing  to  the  development  of  compatible 
Federal,  national,  and  international  standards  in  the  area  of  Coder/Decoders 
(Codecs)  for  Video  Teleconferencing  systems.  It  has  been  prepared  to  inform 
interested  Federal  activities  of  the  progress  of  these  efforts.  Any  comments, 
inputs  or  statements  of  requirements  which  could  assist  in  the  advancement  of 
this  work  are  welcome  and  should  be  addressed  to: 


Office  of  the  Manager 
National  Communications  System 
ATTN:  NCS-TS 


Washington,  DC  20305 
(202)  692-2124 


DTIC 
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SECTION  1 


EXECUTIVE  SUMMARY 


1.1  Backgrouna 

This  document  summarizes  work  performed  by  Delta  Information  Systems,  Inc. 
lUiSJ  for  the  Office  of  Technology  and  Standards  of  the  National  Communications 
System,  an  organization  of  the  U.S.  Government,  under  contract  number 
DcAl 0G-83-C-004 7  Modification  P00004.  The  work  was  performed  under  Subtask  3 
(Performance  of  Codec  Testing  and  Evaluation)  under  Task  3.  Its  known  purpose 
was  to  vigorously  evaluate  the  relative  performace  of  all  1.544  mb/s  video 
teleconferencing  codecs  on  the  market  as  of  November  1984  to  facilitate  the 
choice  of  an  appropriate  frame  format  and  data  compression  algorithm  for  a 
Federal  Telecommunication  Standard  to  meet  the  interoperability  objectives  of 
the  NLS. 

1.2  Scope 

This  report  covers  the  testing  and  ranking  of  motion  codecs  for 
teleconferencing  operating  at  1.544  Mbps.  The  tests  utilized  the  motion  codec 
test  tape  previously  developed  under  Subtask  1  of  Task  3  (Development  of 
Standard  Video  Materials)  and  followed  the  methodology  previously  developed 
under  Subtask  2  of  Task  3  (Development  of  Testing  Methodology).  Codecs 
developed  by  four  manufacturers  were  tested;  this  included  all  units  which  were 
available  as  regular  products  before  the  end  of  November  1984.  Since  the 
results  of  these  tests  are  likely  to  have  considerable  impact  on  the 
manufacturers,  extreme  care  was  taken  to  ensure  absolute  impartiality  and 


fairness  in  the  procedure.  Manufacturers'  representatives  actively  participated 
in  all  testing  and  provided  assurance  that  the  results  were  representative  of 
tne  performance  capability  of  their  equipment.  The  subjective  evaluation  and 
ranking  of  the  codecs  employed  all  possible  safeguards  against  external 
influences  by  test  equipments  or  test  arrangements  and  against  any  personal 
prejudice  of  the  evaluators.  The  test  results  and  codec  rankings  are  absolutely 
firm  without  the  slightest  doubt  or  ambiguity. 

Section  2  of  this  report  describes  the  codec  tests  that  were  performed  at 
locations  selected  by  each  manufacturer.  The  test  tape  was  processed  in  each 
codec  and  the  outputs  recorded  on  1"  tape  for  later  evaluation.  Section  3 
covers  the  subjective  evaluation  and  comparative  scoring  of  the  codec  output 
tapes.  It  details  the  planning  and  implementation  of  the  test  setup,  personnel 
selection,  scheduling,  test  performance  and  initial  review.  Section  4  contains 
the  computation  of  the  numerical  test  results  and  the  tanking  of  the  codecs  and 
several  technical  discussions  of  codec  performance  under  standard  and  special 
conditions  which  back  up  the  numerical  results.  Section  5  gives  a  brief 
conclusion  and  several  recommendations  for  future  test  efforts. 


SECT I ON 2  -  EQUIPMENT  TESTS 


2.1  General 


Initially,  all  known  or  potential  manufacturers  of  motion 
codecs  operating  at  1.544  Mbps  and  using  the  NTSC  standard  were 
solicited  to  establish  equipment  availability  and  interest  in 
participating  in  the  tests.  It  was  stipulated  that  within  the 
expected  time  frame  of  the  tests  ( October /November  1984)  the  codec 
had  to  be  available  as  a  complete  functional  product  line  item. 
Developmental  models  or  units  still  lacking  an  essential  part 
(such  as  the  line  interface  module)  were  not  considered 
acceptable.  This  selection  limited  the  number  of  codecs  to  be 
tested  to  four. 

Great  care  was  taken  to  achieve  complete  impartiality  of  the 
tests.  The  same  copy  of  the  1"  test  tape  was  used  in  all 
instances.  Connection  between  transmitter  output  and  receiver 
input  had  to  be  made  externally;  internal  loop-back  if  available 
was  not  allowed  to  be  used.  Selection  of  the  test  location  was 
left  to  each  manufacturer.  Several  engineers  as  assigned  by  the 
manufacturer  participated  in  the  tests,  and  they  were  given 
unlimited  time  to  verify  that  the  codec  and  the  complete  test 
setup  were  performing  properly.  In  case  even  a  slight  discrepancy 
appeared,  the  test  was  repeated  if  requested  by  the  manufacturer. 
Thus  it  can  be  guaranteed  that  the  tapes  recorded  at  the  codec 
outputs  accurately  represent  the  best  performance  capabilities  of 
each  equipment . 

The  full  length  of  the  test  tape  was  run  through  every  codec. 
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This  includes  46 
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ces  of  motion  and  still 
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nes  for  regular 

performance  eval 
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3  identical  sequences 
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and  13 

test  signals  which  are 

not 

part  of  the 

official  evaluation,  but  ate  expected  to  be  used  later  for  further 
studies.  Furthermore,  the  processed  46  regular  sequences  were  run 
through  the  codec  a  second  tine  to  simulate  a  double  hop;  and  upon 
request  of  the  manufacturer  recordings  of  other  operational  modes 
of  the  codec  were  made . 

Brief  descriptions  of  the  individual  tests  are  subsequently 
presented  in  chronologies  1  order. 

2.2  GEC 

The  GEC  tests  were  performed  in  Philadelphia  where  the  test 
tape  had  been  produced  at  the  TV  studio.  The  test  setup  block 
diagram  is  shown  on  Figure  2-1.  The  full  complement  of  video  tape 
and  monitoring  equipment  was  furnished  by  Center  City  Video.  The 
bit  error  inserter  was  supplied  by  DIS.  In  addition  to  the  test 
tape,  double  hop  testing  was  performed. 

2.3  Fujitsu 

The  tests  on  the  Fujitsu  FEDIS  1.5  equipment  were  performed 
at  t'neir  facility  in  Kawasaki  near  Tokyo.  The  test  setup  is  shown 
on  Figure  2-2.  All  equipment  was  supplied  by  Fujitsu.  A  whole 
test  bay  was  assigned  to  this  fully  integrated  setup.  Several 
engineers  were  present  to  continuously  monitor  equipment 
performance.  Initially  the  performance  with  error  insertion  did 
not  seem  to  come  up  to  theoretical  expectations,  but  after 
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thorough  checking  the  engineers  found  ail  equipment  to  be 
oceracmo  orooeriv.  Only  the  complete  test  tace  was  recorded 
through  the  codec. 

2.4  CLI 

The  tests  on  the  CLI  VTS-1 . 5  codec  were  performed  at  their 
facility  m  San  Jcse,  California.  The  test  setup  is  shown  on 
Figure  2-3 .  The  video  tape  recorders  and  part  of  the  monitoring 
equipment  were  rented  from  a  local  TV  production  studio;  The 
balance  of  the  equipment  was  furnished  by  CLI,  and  the  bit  error 
inserter  by  DI3.  A  unit  used  only  at  CLI  was  the  frame 
synchronizer  which  was  needed  to  process  the  codec  receiver  signal 
before  it  could  be  recorded.  This  is  due  to  the  fact  that  the 
codec  receiver  output  does  not  have  the  horizontal  sync  locked  to 
the  color  subcarrier.  This  condition  does  not  interfere  with 
viewing  the  signal  on  a  monitor  but  is  incompatible  with  tape 
recording . 

Engineers  from  both  CLI  and  the  supplier  of  the  tape 
recorders  participated  in  the  alignment  of  the  test  setup.  Level 
setting,  particularly  through  the  frame  synchronizer,  was  quite 
critical  and  oerformed  with  extreme  care.  To  set  the  proper 
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2  .  5  NEC 

The  tests  on  the  NEC  NETEC-Xl ( MC )  codec  were  performed  at  the 


NTSC  COLOR  MONITOR  SONY  KX-901 

HIT  ERROR  INSERTER  GOR  DLS-106  WITH  INTERFACE  CONVERTER  HAMI  !.>*> 

(IN  CIRCUIT  ONLY  FOR  ERROR  TESTS) 


NEC  America  facility  at  Fairfax,  Virginia.  The  test  setup  is 
shown  on  Figure  2-4.  The  video  tape  recorders  were  provided  by  a 
local  leasing  company,  the  monitoring  equipment  by  NEC  and  the  bi 
error  inserter  by  DIS.  Several  NEC  engineers  were  present  to 
handle  the  lineup  and  operation  of  the  test  setup.  There  were  no 


problems  with  the  codec  proper.  In  addition  to  the  test  tape,  a 
recording  of  double  hop  performance  was  made. 


WAVEFORM 


FIGURE  2-4  TEST  SETU I*  -  NEC 


SECTION  3 


TEST  TAPE  SCORING 


3 . 1  Planning 

The  subjective  evaluation  of  the  test  tapes  which  have  been 
processed  through  the  various  codecs  represents  one  of  the  most 
critical  phases  of  the  overall  program.  The  validity  and 
integrity  of  the  test  results  must  be  guaranteed  beyond  question. 
Not  only  must  the  test  equipment  operate  flawlessly  but  the 
vagaries  of  the  human  element  must  be  eliminated  as  much  as 
possible.  To  satisfy  these  requirements  the  following  factors 
must  be  considered: 

o  Test  setup  and  equipment  check 
o  Personnel  numbers  and  selection 
o  Test  sequence  scheduling 

The  chief  quideline  for  planning  the  test  tape  scoring  is 
CCIR  Recommendation  500-2,  Method  for  the  Subjective  Assessment  of 
the  Quality  of  Television  Pictures,  (Vol.  XI,  Part  1,  XVth 
Plenery  Assembly,  Geneva,  1982). 

3.1.1  Test  Setup 

The  basic  test  requirements  are  satisfied  by  two  1"  video 
tape  recorders,  two  color  monitors,  and  a  double  pole  double  throw 
switch  to  interchange  the  monitors.  Equipment  needed  for  checkout 
must  be  added.  The  monitors  should  have  the  largest  screen  size 
compatible  with  top  quality  performance.  Only  direct  view 
monitors  are  recommended  because  projection  monitors  generally 
suffer  from  some  loss  in  quality  (both  resolution  and  contrast) 
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a~i  a  limited  viewing  angle. 

The  viewing  conditions  should  follow  CCIR  Rec .  500-2  as  a 

guideline.  Most  of  the  parameters  therein  are  stated  in  a  rather 
loose  form  and  do  not  require  close  adherence  which  would  call  for 
elacorate  test  equipment.  Since  the  grading  of  the  codec  pictures 
is  to  he  performed  on  a  comparison  basis,  it  is  mainly  important 
to  ensure  that  both  pictures  will  be  viewed  by  all  obser/ers  under 
essentially  identical  conditions.  This  makes  the  viewing  distance 
a  critical  parameter.  The  recommended  range  is  4-6  times  the 
picture  height  but  a  short  viewing  distance  would  seriously  limit 
the  number  of  possible  observers.  In  practice,  it  seems 
desireable  to  allow  a  very  small  increase  above  the  recommended 
range,  particularly  since  existing  teleconferencing  installations 
have  average  viewing  distances  t  the  practical  maximum  number  of 
simultaneous  viewers  is  5-6. 

3.1.2  Personnel 

Proper  selection  of  the  evaluating  personnel  is  very 
important  to  achieve  valid  and  impartial  results.  CCIR  Rec. 

500-2  and  other  pertinent  publications  distinguish  between  expert 
and  non-expert  observers,  with  the  preference  given  to 
non-experts.  This  recommendation  is  being  followed  completely  and 
is  particularly  applicable  to  the  codec  evaluation.  An  expert  in 
video  coding  would  most  likely  be  able  to  identify  a  codec  from 
the  appearance  of  the  picture  which  would  make  an  impartial 
evaluation  impossible.  It  is  desirable  to  use  observers  who  are 
as  representative  as  possible  of  teleconference  users.  This 
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yields  the  most  meaningful  results.  It  was  decided  to  recruit 
personnel,  both  male  and  female,  from  the  middle  management  level 
of  several  branches  of  activities  in  selected  organizations.  It 
is  unlikely  that  anybody  with  actual  teleconferencing  experience 
would  be  included  but  people  should  be  chosen  who  are  potential 
future  users  of  expanded  teleconferencing  applications. 

The  recommended  range  of  numbers  of  evaluators  is  10  to  20. 
Considering  the  postulated  careful  personnel  selection,  it  is  felt 
that  even  the  minimum  number  (  10  to  12)  will  be  adequate  to 
achieve  dependable  results.  However,  even  this  minimum  number  is 
too  high  to  make  a  tape  comparison  evaluation  in  one  sitting; 
therefore  the  evaluators  will  have  to  be  divided  into  two  groups, 
and  each  test  made  twice. 


3.1.3  Scheduling 


The  test  tape  has  a  running  time  of  slightly  above  30  minutes 
and  contains  49  sequences.  Three  of  these  sequences  are 
intentionally  repeated  for  the  purpose  of  displaying  them  on 
interchanged  monitors  to  determine  if  possible  differences  in 
monitor  quality  may  influence  the  results.  The  first  repeated 
sequence  is  No.  25,  just  in  the  middle  of  the  tape.  Since  4 
processed  tapes  are  to  be  evaluated,  a  total  of  6  different  codec 
pairs  are  to  be  compared.  All  tests  must  be  performed  in  two 
sections  because  of  the  two  groups  of  evaluators,  therefore  12 
separate  tests  are  required.  With  sufficient  allowance  for  setup 
time  and  safety  margin,  one  hour  must  be  scheduled  for  each  test. 
Therefore,  the  tests  require  a  total  period  of  two  days. 


The  following  designations  are  used  on  the  schedule  shown  on 
Table  3-1  : 

Evaluator  groups  :  X,  Y 

Codec  test  tapes  :  1,  2,  3,  4 

Tape  recorders  :  A,  B 

Monitors  :  L,  R  (left,  right) 

This  schedule  was  carefully  prepared  to  eliminate  any 
possible  effects  of  equipment  differences  and  test  timing  in  order 
to  guarantee  a  completely  impartial  evaluation.  The  following 
measures  were  taken  for  this  purpose:  (a)  The  monitors  are 
interchanged  between  the  first  and  second  half  of  each  tape.  The 
assignment  of  each  half  of  the  tape  to  a  monitor  is  reversed 
between  the  two  sections  of  each  test.  (b)  The  assignment  of  each 
tape  to  a  recorder  is  reversed  between  the  two  sections  of  each 
test.  A  tape  is  never  assigned  to  the  same  recorder  on 
consecutive  tests.  Thus  each  tape  is  always  freshly  threaded  into 
a  recorder.  (c)  The  two  sections  of  each  test  are  scheduled  on 
different  days  in  reverse  order.  This  eliminates  possible  effects 
of  time  of  day  and  viewer  fatigue.  The  two  groups  of  viewers  are 
scheduled  alternately  each  every  two  hours,  so  that  each  person 
participates  in  only  3  tests  per  day.  This  is  important  to  avoid 


3.2  I  mp  1  e me "nation 
3.2.1  Test:  settup 

The  schematic  of  the  test  setup  on  Figure  3-1  shows  the 
arrangement  of  the  previously  mentioned  basic  equipment.  In 
addition,  a  color  bar  generator  and  a  distribution  amplifier  are 
available  for  equipment  tests.  the  color  bar  generator  provides  a 
standard  test  signal,  and  the  distribution  amplifier  allows 
displaying  the  same  signal  on  both  monitors  to  ensure  perfectly 
matched  adjustments.  Properly  matched  video  levels  were  verified 
by  means  of  built-in  indicators  on  the  tape  equipment. 

Figure  3-1  gives  the  description  of  all  equipment  units. 

With  the  exception  of  the  relay  switch  used  for  interchanging  the 
monitors  (which  was  built  by  DIS)  all  equipments  were  leased 
locally.  The  tape  recorders  are  the  standard  of  the  industry  for 
portable  operation.  The  monitors  were  a  model  specially  modified 
for  computer  use  which  features  exceptionally  good  resolution 
without  frequency  response  distortion. 

The  comparison  tests  were  performed  in  a  conference  room  in 
Delta  Information  System's  facility  in  Horsham,  Pa.  Figure  3-2 
shows  the  physical  layout.  The  equipment  was  located  on  tables  at 
both  ends  of  the  room.  The  monitor  picture  height  was  15", 
therefore  the  ideal  viewing  distance  was  7  1/2’.  The  front  of 
each  chair  was  located  7'  from  the  center  between  the  monitors 
which  put  the  eyes  of  the  viewers  at  an  average  distance  of  about 
3' .  The  variation  in  viewing  angles  did  not  seem  to  make  make  any 
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The  overhead  lighting  was 


riceable  diff 

reduced  to  provide  about  25  foot  candles  on  the  wall  behind  the 
monitors  and  about  15  foot  candles  at  the  chairs  of  the  observers 
which  gave  just  enough  light  for  marking  the  score  sheets.  A  low- 
level  work  light  on  the  equipment  table  was  mainly  used  for 
threading  the  tape  recorders.  A  screen  prevented  any  light 
reflections  on  the  monitor  fronts  and  also  served  to  separate  the 
operating  personnel  from  the  viewers. 

3.2.2  Personnel 

Since  Delta  Information  Systems  is  located  in  a  modern 
business  center,  several  nearby  companies  provided  sources  for 
suitable  observers  with  minimal  logistics  problems.  The  ten 
people  selected  were  all  of  above  average  intelligence  and 
represented  a  variety  of  backgrounds.  Some  were  engineers;  but 
none  were  experienced  in  the  video  field.  Since  it  is  considered 
important  to  maintain  anonymity  in  the  evaluation  process,  all 
viewers  were  given  numbers  by  which  they  are  identified  for  all 
purposes.  Following  is  their  approximate  job  description  and 
affiliation . 

Group  X: 

1.  Engineering  Manager,  computer  company,  male 

2.  Communication  Engineering  Consultant,  self-employed,  male 

3.  Secretary,  computer  company,  female 

4.  System  Engineer,  computer  company,  male 

5.  Chapter  Officer,  national  civic  organization,  female 
Group  Y: 

6  . 


Marketing  Representative,  communication  systems  company,  male 


Sys ten.  Design  Engineer,  military  systems  company,  male 
3.  Chief  Accountant,  communication  engineering  company,  female 
9.  System.  Design  Engineer,  military  systems  company,  male 
11.  System.  Analyst,  military  systems  company,  female 

3.2.3  Test  Performance 

The  rented  equipment  was  delivered  two  days  before  the 
scheduled  test  date  to  allow  familiarization  and  thorough  checks 
of  the  complete  setup.  A  rental  company  technician  provided 
detailed  instructions  in  the  operation  of  the  tape  recorders  and 
associated  equipment.  The  monitors  had  been  carefully  aligned 
before  delivery  and  required  only  very  minor  adjustments. 
Approximate  compliance  with  CCIR  Rec .  500-2  was  checked  with  a 

light  meter  and  by  visual  observation.  For  the  comparison  tests 
to  be  performed  this  factor  is  less  important  than  the  very  close 
match  of  both  monitors.  This  was  accomplished  visually  by  feeding 
identical  signals,  from  both  the  color  bar  generator  and  the 
unprocessed  test  tape,  to  both  monitors  by  means  of  the 
distribution  amplifier.  These  tests  were  repeated  every  morning. 
The  monitors  proved  to  be  extremely  stable  and  required  very 
little  adjustment. 

Table  3-2  shows  the  test  schedule  which  was  followed  on  the 
two  days  of  the  tests.  Initially  all  ten  evaluators  were 
assembled  for  an  introduction.  The  instructions  included  in 
Appendix  A  together  with  a  sample  score  sheet,  had  all  previously 
been  given  out  to  the  observers.  These  were  reviewed  in  detail 
during  the  introductions. 
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TABLE  3-2  TEST  PERFORMANCE  SCHEDULE 


rne  original  unprocessed  video  rape  was  shown  or.  both 
monitors  to  give  the  evaluators  an  idea  what  kind  of  pictures  they 
will  have  to  judge.  Special  suggestions  were  given  recardmc  the 
sequences  with  error  insertion.  Questions  for  clarifications  were 
solicited  but  none  were  forthcoming. 

The  test  schedule  was  followed  accurately  throughout  both 
days.  No  problems  were  encountered.  Care  had  to  be  taken  in 
cueing  up  and  starting  the  tapes  so  they  were  synchronized  to  a 
time  difference  of  well  under  1/4  second.  Any  larger  shift 
between  the  two  pictures  proved  very  annoying  and  would 
undoubtedly  affect  the  accuracy  of  scoring.  Once  started,  the  two 
tape  recorders  maintained  synchronism  for  the  length  of  the  tape. 


3.3  Raw  Data  Review 


Table  3-3  shows  one  sample  of  the  60  two-page  score  sheets 
which  were  filled  in  by  the  evaluators.  It  should  be  noticed  that 
these  sheets  are  completely  anonymous.  The  evaluator  is 
identified  only  by  a  number,  and  so  is  the  test  which  is  being 
scored.  The  codec  numbers  could  be  derived  from  the  test  numbers 
only  by  use  of  Table  3-1  which  was  not  available  to  the 
evaluators.  The  assignment  of  numbers  to  each  codec  was  at  that 
time  known  only  to  a  single  person.  No  written  record  was  made. 
Thus  the  evaluators  could  not  obtain  any  information  on  the 
equipment  under  test  and  their  impartiality  can  be  firmly 
guaranteed . 

The  score  sheet  grades  were  translated  into  numerical  scores 
by  equating  M=3,  3=2,  S=l.  The  sign  of  the  score  was  determined 


TABLE  3-3  TYPICAL  FILLED-IN  SCORE  SHEET 


by  stipulating  a  positive  number  in  favor  of  the  lower  numbered 
codec,  and  a  negative  number  in  favor  of  the  higher  numbered 
codec.  In  order  to  facilitate  transcribing  the  scores  which  are 
referenced  only  to  monitor  locations,  the  numbers  of  the  2  codecs 
under  test  together  with  the  applicable  sign  of  the  score  were 
added  to  each  score  sheet  after  completion  by  the  evaluator.  The 
reversal  after  sequence  24  is  obvious . 

A  quick  review  of  the  score  sheets  shows  fairly  wide 
variations  between  evaluators  which  was  to  be  expected  due  to  the 
subjectivity  of  the  tests.  Some  evaluators  used  mainly  "equal" 
and  "slightly  better"  while  others  made  frequent  use  of  "much 
better"  and  "better".  However,  in  spite  of  these  variations  the 
general  tendency  of  the  scores  is  very  consistent.  The  variations 
show  no  apparent  correlation  between  seating,  monitor  locations 
and  scores.  The  results  of  the  monitor  cross  checks  by  means  of 
repetition  of  sequences  11,  15  &  16  are  shown  on  Table  3-4  which 
lists  the  scores  of  each  original  and  repeated  sequence  given  by 
all  evaluators.  Agreement  is  far  from  perfect  but  most 
differences  (71  out  of  180  cross  checks  or  39.4%)  are  only  by  one 
point  which  is  an  expected  margin  of  error  for  a  subjective  test. 

There  are  only  10  differences  (5.6%)  of  2  points,  one  (.6%) 
points,  and  3  (1.7%)  reversals  from  +1  to  -1.  Larger  numbers  and 
size  of  differences  seem  to  go  with  some  codec  pairs  and 
evaluators.  There  is  no  consistency  in  the  direction  of  the 
differences  which  might  indicate  a  preference  for  one  of  the 

monitors.  This  was  the  main  purpose  of  the  repetition  of  > 

sequences  which  has  thereby  proven  its  value. 
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TABLE  3-4  MONITOR  CROSS-CHECK  SCORES 


EVALUATOR  SCORES 


TEST 

TAPE 

SEQU¬ 

ENCES 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11/25 

+  1/  +  2 

+2/  +  3 

+1/  +  3 

+  1/  +  1 

+2/  +  2 

+2/+1 

+1/  +  2 

+  1/+1 

+2/  +  3 

+  1/+2 

1-2 

15/35 

+  2/  +  2 

+  1/  +  2 

+2/ +3 

+3/  +  2 

0/+1 

+3/ +2 

+  2/  +  2 

+2/  +  2 

+3  /  +  3 

+1/  +  3 

16/26 

0/  +  2 

+  1/+3 

+2/  +  3 

+2/  +  1 

+3/+3 

+  2/+1 

+  1/+1 

+1/+1 

+3  / 4  3 

+  2  /  +  2 

11/25 

-1/  +  1 

-1/0 

0/-1 

0/0 

0/-1 

0/-1 

0/-1 

-1/0 

0/  0 

0/0 

1-3 

15/35 

-1/-1 

-1/-1 

0/0 

0/0 

-1/+1 

0/-1 

-1/-1 

0/-1 

-1/0 

-1/0 

16/26 

-1/-1 

-1/0 

0/0 

0/0 

0/0 

0/-1 

-1/-1 

-1/0 

-1/0 

-1/0 

11/25 

+  3/0 

+  2/  +  2 

+1/+1 

0/0 

-1/0 

-1/-1 

0/+1 

0/0 

-1/0 

+  1/  +  1 

1-4 

15/35 

0/-2 

0/-1 

0/0 

-1/-1 

0/0 

-2/0 

-1/-1 

0/-1 

-1/-1 

-1/-1 

16/26 

-21-2, 

0/0 

+  1/+1 

-1/-1 

0/+1 

-1/0 

-1/-1 

0/-1 

-1/-1 

0/  -1 

11/25 

-1/-3 

~  3  /  —  3 

-3/-3 

-2/-2 

-1/-1 

-1/-1 

-2/-1 

-2/-2 

-2j-2 

-2/ -2 

2-3 

15/35 

—  3  /  —  2 

—  3  /  —  3 

—  3  /  —  3 

-3/-3 

—  3  /  —  3 

-2/-2 

-2/-2 

-21-2 

-21-2 

-21-2 

16/26 

-3/-3 

-3/-3 

— 3  /  —  3 

-2/-2 

—  3  /  —  2 

-2/-1 

-1/-1 

- 21-2 

-3 1-2 

-2/-1 

11/25 

-1/-2 

-2/-2 

-1/-1 

-1/-1 

-2/0 

-2/-1 

-2/-2 

-2/-1 

-21-2 

-2/-1 

2-4 

15/35 

-3/-2 

— 3  /  —  3 

—  3  / — 3 

—  3 /  — 3 

-2/-3 

-2/-2 

-2/-3 

—  3 / — 3 

-21-2 

—  3  /  —  2 

16/26 

-3/-3 

-3/-3 

-3/-3 

-3/-2 

-3/-2 

-3/-2 

-3/-2 

-2/ -2 

-2/ -2 

-2/0 

11/25 

+  1/  +  1 

+2/  +  2 

+  2/  +  2 

0/  +  1 

+  1/  +  2 

-1/0 

+  1/+1 

+1/-1 

+1/+1 

+1/  +  1 

3-4 

15/35 

+  1/0 

+  1/0 

+  1/0 

0/0 

+  2/0 

-1/+1 

+1/+1 

+1/+1 

+1/  +  1 

0/0 

16/26 

+  2/  +  1 

0/0 

+  1/+1 

-1/-1 

0/0 

-1/0 

+  1/+1 

+1/+1 

0/0 

+  1/0 

3  -  15 


SECTION'  4 


TEST  RESULTS 


4.1  Numerical  Scores  and  Ranking 

The  60  score  sheets,  a  typical  sample  of  which  was  shown  on 
Table  3-3,  were  collated  into  6  groups,  one  for  each  codec  pair 
under  test.  Subsequently,  the  marked  scores  were  translated  into 
numbers  from  t3  to  -3  and  entered  in  matrix  form  into  a  computer 
as  shown  on  Tables  4-1  to  4-6.  Mean  values  and  standard 
deviations  were  computed  for  each  line  (test  sequence)  and  column 
(evaluator).  The  mean  of  either  line  or  column  mean  values  is  the 
grade  or  comparitive  score  between  the  two  codecs  and  is  circled 
on  each  table. 

The  standard  deviation  values  are  not  used  directly  for 
scoring  but  only  to  check  the  validity  of  the  scores.  If  one 
particular  test  sequence  showed  a  high  standard  deviation  for 
several  codec  pairs,  it  would  be  an  indication  that  this  sequence 
presents  serious  problems  and  should  possibly  be  eliminated  from 
the  computation  of  the  comparative  scores.  Similarly,  any 
evaluator  with  a  mean  score  much  different  from  the  others  in 
several  instances  must  be  considered  suspect  and  should  possibly 
be  eliminated.  However,  close  scrutiny  of  the  results  confirms 
that  all  scores  are  fully  valid.  Most  standard  deviations  of  test 
sequence  scores  are  well  below  1.00,  and  the  few  higher  values  are 
randomly  scattered.  The  variations  of  the  mean  scores  of  the 
evaluators  are  caused  by  differences  of  opinion  regarding  the 
annoyance  of  different  degradations  and  are  within  expected 
limits.  The  standard  deviations  of  the  evaluator  scores  mainly 
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reflect  the  tendency  to  use  "much  better"  and  "better"  scores  more 
or  less  frequently  but  a  high  value  can  also  indicate  very  erratic 
scoring . 

When  looking  at  the  mean  scores  of  each  evaluator  on  Tables 
4-1  to  4-6,  it  is  interesting  to  note  that  in  5  of  the  codec 
comparisons,  the  scores  are  unanimous.  Only  in  the  case  of  GEC 
vs.  NEC  is  there  a  split  decision  but  the  majority  and  the 
numerical  values  clearly  favor  GEC. 

Table  4-7  summarizes  the  individual  results.  The  ranking 
matrix  gives  the  mean  of  the  scores  of  each  codec  against  the  3 
others  and  results  in  the  following  ranking. 

Model  No. 

1.  CLI  VTS-1.5  E 

2.  GEC  525  Line  (1.5/0.77  MBIT/S) 

3.  NEC  NETEC-X1  (MC) 

4.  Fujitsu  FEDIS  1.5  (B) 

The  score  comparison  chart  shows  graphically  both  the  relative 
scores  and  the  resulting  mean  values.  Obviously,  this  chart  is 
not  to  scale  and  the  various  numbers  cannot  add  up  because  they 
represent  means  derived  in  different  steps  from  independent 
subjective  scores.  The  value  of  the  chart  is  that  it  shows  full 
consistency  of  the  results.  All  score  differences  are  in  the  same 
direction,  there  is  no  reversal  which  would  indicate  a  possible 
ambiguity.  Thus  the  validity  of  the  final  ranking  can  be  firmly 
guaranteed . 

To  get  further  data  regarding  the  most  critical  No.  1  and  2 
ranks,  three  experts  performed  the  comparison  scoring  test. 
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This  fully  confirms 

the  of 

f  icial 

score.  As  a 

atter  cf  fact, 

,  the  numerical  value 

1  s  tw  u 

ce  the 

official  score  in 

favor  cf  C LI  which  is  caused  by  the  fact  that  experts  tend  to  use 
high  level  scores  (such  as  "much  better")  more  freely  than 
tor. -experts  . 

4.2  Discussion  of  Degradations 

The  numerical  test  results  are  fully  consistent  and  need  no 
further  justification.  However,  it  is  desirable  to  perform  a 
brief  technical  analysis  of  the  degradation  caused  by  each  codec 
to  establish  a  rationale  behind  the  subjective  scores.  The 
various  evaluators  obviously  had  different  opinions  regarding  the 
impact  of  each  degradation. 

The  most  noticeable  degradation  that  can  be  caused  by  the  CLI 
codec  is  a  series  of  spurious  colored  squares  or  stripes.  They 
appear  only  very  infrequently,  and  seem  to  be  caused  by  certain 
picture  conf igurations .  They  do  not  seem  to  be  a  random 
■occurrence  since  they  happened  several  times  at  the  same  spots  on 
the  tape.  Some  motion  has  a  slightly  jerky  appearance.  There  is 
a  slight  color  shift  towards  yellow  which  may  be  due  either  to  the 
codec  proper  or  to  a  slight  misaa justment  in  the  frame 
synchronizer  which  had  to  be  inserted  in  order  to  make  the  codec 
output  compatible  with  the  tape  recorder.  In  normal  operation 
this  unit  is  not  needed,  the  regular  codec  output  can  be  connected 
to  any  color  monitor  or  to  the  input  of  another  codec. 

The  GEC  codec  suffers  most  obviously  during  scenes  with 
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moderate  to  agitated  notion  which  causes  severe  tearout  in  the 
bottom  half  of  the  picture.  Some  of  the  sane  problem  is  also 
noticeable  after  switching.  Edges  become  quite  ragged  during 
notion.  Furthermore,  any  type  of  motion  including  switching 
leaves  a  coarsely  grained  after  image  for  about  1  to  3  seconds. 
This  image  disappears  in  a  downward  wipe. 

The  most  disturbing  effect  of  the  NEC  codec  is  the  formation 
of  spurious  contour  patterns,  largely  in  the  background,  with  any 
motion,  at  times  even  in  a  still  picture.  Any  slight  change  or 
movement  in  the  picture  causes  a  change  in  this  pattern  which 
makes  it  much  more  noticeable.  A  moving  object  becomes  blurry. 
There  is  a  small  but  noticeable  loss  of  chroma  amplitude  through 
the  codec . 

The  Fujitsu  codec  has  one  deficiency  which  overshadows  all 
other  problems.  It  evidently  can  process  only  a  limited  amount  of 
change  between  successive  frames.  When  this  limit  is  reached  the 
codec  simply  stops  processing  the  picture  change  and  displays  the 
contents  of  the  previous  frame.  This  produces  one  or  a  series  of 
wide  horizontal  stripes  containing  portions  of  one  or  several 
previous  frames  in  the  bottom  of  the  picture.  This  effect  is 
annoying,  and  also  destroys  picture  information.  In  addition, 
edges  become  very  ragged  in  motion,  and  some  blurring  and  spurious 
contours  are  also  noticeable.  A  switch  between  pictures  appears 
like  a  vertical  wipe. 

4.3  Performance  under  Error  Conditions 


Sequences  47,  48,  and  49  on  the  test  tape  are  identical  and 


were 


m 


Y 


used  wit’ii  inserted  error  rates  of  10-5,  10-4,  and  10-3 
respectively,  which  of  course  causes  the  output  image  to  degrade 
to  varying  degrees.  Thus  performance  conditions  is  included  in 
the  general  evaluation  but  with  the  low  weighting  of  3/49  or  abou 
6%.  This  is  fully  justifiable  since  most  transmission  circuits 
are  good  and  error  conditions  occur  only  very  infrequently. 

Scoring  of  the  sequences  with  errors  proved  to  be  somewhat 
difficult  for  the  non-expert  evaluators  because  they  do  not  have 
the  knowledge  to  separate  the  effect  of  errors  from  other  picture 
degradation  and  it  was  not  feasible  to  give  them  specific 
instructions  in  this  matter.  At  a  low  error  rate  (lO-^)  the 
degratation  due  to  errors  may  be  so  slight  that  the  score  was 
based  on  picture  differences  without  errors  which  was  not 
intended.  On  the  other  hand,  at  a  high  error  rate  (10“3)  the 
codec  may  cease  functioning  and  go  into  a  freeze-frame  mode 
showing  a  still  picture  without  disturbance  which  probably  would 
be  scored  higher  than  an  unrecognizable  picture  even  though  both 
are  equivalent  and  indicate  a  complete  codec  malfunction. 

However,  a  brief  discussion  of  performance  under  error 
conditions  based  on  "expert"  observations  is  desirable  in  case  it 
is  ever  necessary  to  use  a  codec  over  a  considerably  degraded 
transmission  circuit. 

The  error  test  sequences  consist  of  two  scenes,  one  with 
slight  and  one  with  moderate  motion.  The  influence  of  error  is 
decidedly  more  pronounced  with  increased  motion. 

At  an  error  rate  of  10-5,  NEC  shows  no  noticeable 
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degradation.  This  is  due  to  the  fact  that  a  more  powerful  forward 
error  correction  code  is  used  than  in  the  other  units.  GEC  shows 
some  small  colored  dots  and  very  few  short  colored  streaks.  CLI 
has  occasional  colored  bands  accross  the  picture.  Fujitsu  suffers 
from  an  occasional  complete  loss  of  picture. 

Performance  at  an  error  rate  of  10“^  is  somewhat  similar  but 
more  deteriorated.  NEC  shows  occasional  colored  blotches  and  a 
few  streaks  but  the  overall  picture  is  only  very  slightly 
disturbed.  The  colored  dots  and  streaks  in  the  GEC  codec  are  more 
frequent  and  the  streaks  much  longer  but  the  degradation  is  not 
serious.  CLI  shows  severe  disturbance  by  wide  colored  bands  which 
at  times  almost  obliterate  the  picture.  Fujitsu  has  complete 
picture  loss  for  a  large  portion  of  the  time. 

An  error  rate  of  10“^  is  very  high  and  digital  equipments 
cannot  be  expected  to  operate  properly.  However,  the  GEC  codec, 
though  severely  disturbed  by  long  colored  streaks,  maintains  a 
recognizable  picture  with  sync  and  color  intact  can  be  considered 
marginally  usable.  All  other  units  produce  either  freeze  frame  or 
complete  picture  loss  and  thus  are  called  totally  unusable. 

4.4  Double  Hop  Performance 

Though  not  required,  it  was  deemed  valuable  to  unofficially 
evaluate  "double  hop"  (tandem)  performance  of  codecs,  using  the 
output  tape  processed  through  each  codec  as  input  and  making  a 
second  recording.  Time  limitations  made  it  impossible  to  do  this 
at  Fujitsu.  It  is  anticipated  that  there  will  be  occasions  of 
either  long  haul  transmissions  over  two  satellite  links  or  usage 
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in  various  networks  where  a  picture  will  have  to  be  processed 
through  two  codecs  in  tandem,  and  it  was  assumed  that  both  will  be 
of  the  same  type. 

A  very  limited  subjective  evaluation  was  performed  by  three 
non-expert  DIS  employees.  The  two  comparison  scores  that  were 
obtained  and  are  shown  on  Table  4-8  fully  confirm  the  ranking 
developed  in  4.1  As  a  matter  of  fact,  the  difference  scores  of 
CLI  vs  GEC  (.75)  and  GEC  vs  NEC  (1.07)  are  so  much  more  pronounced 
than  those  in  4.1  that  a  comparison  test  between  CLI  and  NEC  would 
not  have  added  any  useful  information. 

It  has  thus  been  established  that  adding  a  requirement  of 
double  hop  performance  would  have  no  effect  on  the  previously 
developed  ranking  of  the  codecs.  Following  is  a  short  discussion 
of  the  effect  of  tandem  operation  of  each  codec  to  give  a 
rationale  behind  the  subjective  ranking  which  is  based  on  limited 
data  . 

In  the  CLI  codec,  the  spurious  colored  squares  and  stripes 
remain  essentially  unchanged.  The  jerky  motion  becomes  more 
pronounced,  and  a  little  blurring  is  added.  The  color  shift 
towards  yellow  becomes  more  noticeable  but  may  have  been  caused  by 
the  frame  synchronizer  and  not  be  inherent  to  the  codec.  Overall, 
the  additional  degradation  caused  by  the  double  hop  is  not  severe. 

The  main  degradation  in  the  GEC  codec,  namely  the  tear  out  in 
the  bottom  half  of  a  picture  with  motion,  is  only  slightly 
increased.  The  ragged  edges  during  motion  become  more  pronounced, 
and  occasionally  may  wipe  out  some  picture  information.  The  most 
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0 

24 

0.7  0.58 

1 

0 

1 

25 

1.3  0.58 

2 

1 

1 

26 

*0.3  1.15 

-1 

-1 

1 

27 

1.0  1.00 

1 

0 

2 

28 

1.0  1.73 

0 

0 

T 

29 

1.7  0.58 

1 

"> 

2 

30 

1.7  0.58 

1 

2 

2 

31 

1.7  0.58 

1 

4 

,2 

32 

1.3  0.58 

1 

1 

2 

33 

0.7  0.58- 

1 

1 

0 

34 

0.7  0.58 

1 

1 

0 

35 

1.0  1.00 

2 

1 

0 

34 

1.3  0.58 

1 

1 

2 

37 

1.0  1.00 

0 

1 

2 

38 

1.7  0.58 

2 

1 

2 

39 

0.7  0.58 

1 

1 

0 

40 

0.0  1.00 

-1 

1 

0 

41 

-0.7  0.58 

-1 

0 

-1 

42 

2.0  1.00 

2 

1 

3 

43 

1.3  1.15 

0 

2 

2 

44 

1.0  0.00 

1 

1 

1 

45 

1.7  0.58 

1 

2 

2 

46 

1.3  0.58 

1 

1 

2 

MEAN 

-0.93 

-0.59 

-0.74 

STD. DEV. 

1.12 

0.96 

1.00 

SRAOE 

-0.75 

TABLE  4-8 

4  - 


nE  AN 

1.15 

0.98 

1.09 

STD. DEV. 

0.92 

0.30 

0.96 

SPADE 

1.07 

DOUBLE  HOP  SCORFR 
15 


noticeable  effect  of  the  double  hop  is  in  the  coarsely  grained 
after  image  which  becomes  much  more  pronounced  and  is  retained 
over  a  longer  period. 

The  NEC  codec  is  severely  affected  by  the  moving  spurious 
contour  patterns  which  result  in  the  processed  picture  containing 
much  more  motion  than  the  input.  When  this  picture  is  processed  a 
second  time,  the  codec  cannot  listinguish  between  real  and 
spurious  motion  and  a  large  amount  of  contour  patterns  is  added. 
This  problem  is  emphasized  by  tandem  operation  on  a  much  more  than 
linear  scale  and  completely  overshadows  any  other  deterioration. 
Picture  information  is  often  obliterated  and  overall  performance 
is  largely  unusuable. 


SECTION  5  CONCLUSION  AND  RECOMMENDATIONS 


5 . 1  Conclusion 

The  testing  of  full  motion  codecs  at  1.544  Mbps,  which  were 
available  as  finished  products  before  the  end  of  November  1984, 
has  been  successfully  completed.  The  previously  developed 
standard  test  tape  was  fed  through  the  available  codecs  at 
locations  selected  by  the  manufacturers.  The  output  was  recorded 
only  after  agreement  by  the  manufacturer's  engineer  that  it 
represented  the  best  performance  of  the  codec.  The  output  tapes 
were  submitted  to  an  impartial  panel  of  evaluators  for  comparison 
scoring.  Extreme  care  was  taken  to  ensure  that  the  evaluators 
would  remain  completely  unbiased,  and  that  the  scores  could  in  no 
way  be  affected  by  accidental  differences  in  monitoring  and  test 
equipment  or  any  other  extraneous  influences. 

The  results  that  were  computed  from  the  "raw"  scores  of  the 
evaluators  are  fully  consistent  with  each  other.  There  are  no 
contradictions  or  ambiguites  that  would  have  to  be  explained,  the 
numbers  speak  for  themselves  and  need  no  interpretation.  A 
technical  review  by  experts  reached  the  same  results.  Thus  the 
resulting  ranking  of  the  codecs  is  absolutely  firm. 

It  must  be  emphasized  that  the  purpose  of  the  test  is 
strictly  the  comparative  evaluation  of  the  various  compression 
algorithms.  Other  factors,  such  as  size,  weight,  ease  of 
operation,  special  features,  power  consumption,  reliability, 
maintenance  requirements,  cost  and  others  become  important  in  the 
selection  of  a  specific  equipment.  However,  the  statement  of  thi 


cask  does  not  include  consideration  of  any  of  these  factors. 


5.2  Recommendations 

5.2.1  Objective  Tests 

The  subjective  test  results  of  this  task  were  all  produced 
with  the  main  part  of  the  motion  codec  test  tape.  Recordings 
through  all  codecs  were  also  made  of  the  subsequent  portion  of  t  le 
tape  containing  the  standard  signals  for  conventional  analog  tests 
and  also  signals  that  show  promise  for  objective  measurement  of 
motion  performance.  It  is  recommended  to  compare  the  results  of 
the  subjective  evaluations  with  the  objective  measurements  and  to 
determine  the  degree  of  correlation  between  them.  This  will 
require  the  use  of  a  full  capablity  1"  tape  recorder  with  still 
frame  and  single  frame  advance  capabilty.  This  is  not  possible 
with  portable  recorders,  therefore  such  tests  will  have  to  be 
performed  in  a  recording  studio  where  other  test  facilities  are 
readily  available.  The  expected  result  is  a  simplified 
methodology  for  motion  codec  testing  which  will  reduce  and  maybe 
ultimately  eliminate  the  need  for  subjective  evaluation. 

5.2.2  Updated  Equipment  Tests 

Any  test  program  in  a  rapidly  developing  technical  area  has 
the  problem  of  achieving  up-to-date  results.  It  is  impossible  to 
wait  for  every  improved  equipment  which  is  just  about  a  finished 
product  because  the  program  could  never  be  completed.  Therefore 
November  1984  had  to  be  established  as  the  cutoff  date,  with  the 
full  realization  that  new  equipments  would  be  available  within  a 

5-2 
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'-.’STRUCT IONS  FOR  CODEC  PICTURE  EVALUATORS 


The  task  of  these  tests  is  to  judge  tne  comparative  quality 


>f  two  TV  pictures.  There  is  no  absolute  way  to  measure  this 


quality.  Your  judgement  depends  entirely  on  your  personal 


.mpressicn  and  preference.  The  pictures  you  will  have  to  judge 


are  generally  of  poorer  quality  than  the  ones  you  are  accustomed 


to  see  on  your  home  receivers. 


These  tests  pertain  to  the  application  of  TV  for 


ce leconf erencing  which  means  the  distant  participants  of  the 


conference  are  seen  on  a  monitor.  This  does  not  require  the  high 


picture  quality  of  entertainment  TV.  Therefore,  the  picture  is 


orocessed  to  allow  lower  cost  transmission  at  some  sacrifice  in 


picture  quality.  Several  processing  equipments  are  available  all 


of  which  produce  different  types  and  degrees  of  picture 


degradation.  The  purpose  of  this  program  is  to  determine  which 


equipment,  produces  the  most  acceptable  picture  for  the 


teleconference  user.  You  should  put  yourself  in  the  position  of 


the  person  looking  at  the  pictorial  information  as  if  it  were 


transmitted  from  the  distant  conference  room. 


On  the  two  monitors  in  front  of  you,  you  will  see  the  same 


picture  but  processed  through  two  different  equipments  and 


therefore  generally  showing  different  degradations.  The  score 


sheets  in  your  hands  allow  you  to  indicate  if  you  consider  the 


picture  on  either  the  left  hand  or  right  hand  monitor  slightly 


better,  better,  or  much  better  than  the  other,  or  if  you  find  both 


pictures  of  equal  quality.  You  simply  have  to  check  the 


A  -  1 
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applicable  block.  The  difference  between  the  three  grades  of 
"beater"  depends  on  your  subjective  judgement.  It  is  not  possible 
to  list  all  potential  types  of  degradations,  but  following  are 
examples  of  what  you  are  likely  to  encounter: 
o  Loss  of  resolution  (sharpness) 
o  Bad  or  distorted  colors 
o  Moving  parts  of  picture  fuzzy 
o  Moving  parts  of  picture  broken  up 
o  Jerky  motion 

o  Spurious  lines  and/or  other  patterns,  mainly  in  the 
background 

o  Picture  torn  up  and/or  full  of  streaks 
o  Portions  of  more  than  one  picture  on  screen  at  the 
same  time 

The  pictures  to  be  judged  are  recorded  on  video  tape. 
Following  the  title,  the  main  portion  of  the  tape  consists  of  46 
sequences,  each  averaging  less  than  30  seconds  in  length,  followed 
by  10  seconds  long  numbered  scoring  intervals  which  give  you  time 
to  consider  and  record  your  score.  The  sequences  are  an  arbitrary 
mixture  of  still  and  many  kinds  of  moving  pictures,  mostly  in 
color.  Some  sequences  consist  of  one  continuous  scene,  others 
contain  two  or  more  different  scenes.  Your  score  should  reflect 
your  integrated  impression  of  the  whole  sequence,  not  merely  of 
one  portion  such  as  the  end. 

In  the  course  of  the  tests  you  will  see  the  same  degraded 
sequences  several  times.  There  will  even  be  repeats  within  the 
same  test.  Please  score  each  sequence  exclusively  by  your 
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